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Executive Summary 

As part of Kennedy Space Center’s (KSC’s) challenge to open the space frontier, new spaceport 
technologies must be developed, matured and successfully transitioned to operational systems. 
R&D investment decisions can be considered from multiple perspectives. Near, mid and far term 
technology horizons must be understood. Because a multitude of technology investment 
opportunities are available, we must identify choices that promise the greatest likelihood of 
significant lifecycle benefits. At the same time, the costs and risks of any choice must be well 
understood and balanced against its potential returns. The problem is not one of simply rank- 
ordering projects in terms of their desirability. KSC wants to determine a portfolio of projects 
that simultaneously satisfies multiple goals, such as getting the biggest bang for the buck, 
supporting projects that may be too risky for private funding, staying within annual budget 
cycles without foregoing the requirements of a long term technology vision, and ensuring the 
development of a diversity of technologies that support the variety of operational functions 
involved in space transportation. 

This work aims to assist in the development of methods and techniques that support strategic 
technology investment decisions and ease the process of determining an optimal portfolio of 
spaceport R&D investments. Available literature on risks and returns to R&D is reviewed and 
most useful pieces are brought to the attention of the Spaceport Technology Development Office 
(STDO). KSC’s current project management procedures are reviewed. It is found that the “one 
size fits all” nature of KSC’s existing procedures and project selection criteria is not conducive 
to prudent decision-making. Directions for improving KSC’s procedures and criteria are 
outlined. With help of a contractor, STDO is currently developing a tool, named Change 
Management Analysis Tool (CMAT)/ Portfolio Analysis Tool (PAT), to assist KSC’s R&D 
portfolio determination. A critical review of CMAT/PAT is undertaken. Directions for the 
improvement of this tool are provided. STDO and KSC intend to follow up on many, if not all, 
of the recommendations provided. 
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Methodology Development for Assessment of 
Spaceport Technology Returns and Risks 

Prafulla Joglekar 


1. INTRODUCTION 

As part of Kennedy Space Center’s (KSC’s) challenge to open the space frontier, new spaceport 
technologies must be developed, matured and successfully transitioned to operational systems. 
R&D investment decisions can be considered from multiple perspectives. Near, mid and far term 
technology horizons must be understood. Because a multitude of technology investment 
opportunities are available, we must identify choices that promise the greatest likelihood of 
significant lifecycle benefits. At the same time, the costs and risks of any choice must be well 
understood and balanced against its potential returns. The problem is not one of simply rank- 
ordering projects in terms of their desirability. KSC wants to determine a portfolio of projects 
that simultaneously satisfies multiple goals, including getting the biggest bang for the buck, 
supporting projects that may be too risky for private funding, staying within annual budget 
cycles without ignoring the requirements of a long term technology vision, and ensuring the 
development of a diversity of technologies that support the variety of operational functions 
involved in space transportation. 

This work aims to assist in the development of methods and techniques that support strategic 
technology investment decisions and ease the process of determining an optimal portfolio of 
spaceport R&D investments. Towards that goal, this summer, I pursued three different paths. 
First, I reviewed available literature on risks and returns to R&D. My findings and 
recommendations from this review are summarized in Section 2. Second, I reviewed KSC’s 
current project management procedures in terms of their ability to support prudent R&D project 
selection. The findings from that review are summarized in Section 3. Third, I undertook a 
critical review of a tool, named Change Management Analysis Tool (CMAT)/ Portfolio Analysis 
Tool (PAT), currently being developed by Knowledge Based Systems, Inc. (KBSI) under a 
contract from KSC. My findings and recommendations from that review are summarized in 
Section 4. The final section summarizes the recommendations from all three paths of my activity. 

2. A REVIEW OF RELEVANT LITERATURE 

I identified, gathered, browsed through numerous books and articles on such topics as: 

• Technology Management [1], [5], [12], [13], [16] 

• R&D Project Selection [2], [4], [9], [11] 

• Cost-Benefit Analysis [6], [7], [8], [10], [15] 

• Analysis of Public Decisions [3], [14] 

• Risk Assessment and Decision Making [2], [4], [9], [1 1], [13] and 

• Measuring Technology Investment Payoffs [2], [3], [5], [10], [11], [13], [15], 

Several pieces in the general R&D literature had references to space transportation R&D. 
However, I could not locate a single piece that was exclusively devoted to spaceport R&D 
project selection. Although more focused on industry rather than government, the available 
literature is vast. I have limited the references only to the works that may be most useful to 
STDO/ KSC. I would recommend that project selection decision-makers at KSC/ STDO should 
familiarize themselves with at least one or two works in each of the topic areas listed above. 
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Although there are a few reports of successful practical applications, most of the literature is 
theoretical. On the other hand, as most personnel at KSC (a science focused organization) would 
agree, there is nothing more practical than a good theory. The literature that I have cited is 
conceptually very rich and quite applicable to KSC’s project development and portfolio 
selection problem. 

In particular, Martinofll] and Brandenburg [2] seem most relevant, for the following reasons: 

S KSC has been using scoring models in all kinds of decision areas for quite some time now. 
These books provide a better understanding of theoretical underpinnings of scoring models 
and explain proper ways of developing scoring schemes. 

■S Martino ([11], Ch. 5) presents a mathematical programming (constrained optimization) 
model that may be most applicable to KSC’s R&D portfolio selection problem since it is best 
suited to deal with the multiplicity of goals KSC wants to accomplish, such as: 

• Fund projects with the greatest likely benefits, 

• Fund projects too risky for private investment, 

• Fund projects that are adequately diverse in terms of: 

o the Spaceport Technology Development Initiative (STDI) areas supported 
o the TRL levels they seek to accomplish 
o risky and not-so-risky projects 

o old (on-going) projects and new projects Comply with the annual budget cycles and 
still have a long-term perspective on the technologies funded, 

• Provide well-structured justification for growth in KSC’s R&D budget in competition 
with NASA’s traditional R&D centers, 

• Within each Spaceport Transportation Development Initiative (STDI) area, ensure that 
available KSC employees and contractor/ university personnel can meet the FTE 
requirements of funded projects. 

A constrained optimization model also allows one to accommodate constraints such as: 

• Maximum budget to be spent, year by year 

• Maximum funds to be spent on projects in specific functional areas (e.g., fluid systems, 
or command and control, etc.) 

• Minimum % of total spending to go to specific STDI 

• Maximum number of projects to be funded 

Finally, through a properly done sensitivity analysis, a constrained optimization model can 
give us valuable information about the robustness of an optimal solution and about the 
impact of relaxing certain constraints or changing certain objectives. 


3. A REVIEW OF KSC’ S PROJECT MANAGEMENT PROCEDURES 


On KSC’s internal website, I located numerous documents that describe KSC’s “authorized” 
procedures, practices, and data requirements for R&D project formulation, review, selection and 
management. Below is a partial list of those documents. 


o KDP-KSC-P-27 64 
o KDP-KSC-P-2755 
o KDP-F-5002 
o KDP-KSC-P-2602 


Program/ Project Management Process 
Initial Project Formulation 
Research Project Plan 
Project Approval 
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o KDP -P-1453 KSC Independent Assessment Process 
o KDP-P-5016 SE&T Project Management Process 

A careful study of these documents suggests that KSC is interested in using a systematic and 
uniform process for managing all R&D projects. While this approach helps seemingly “fair and 
equitable” practice of such administrative functions as budget administration, project 
documentation, and auditing, I believe its “one size fits all” nature is not conducive to prudent 
project/ portfolio selection. Uniformity requires that the process, the data requirements, and the 
compliance expectations must be simple enough to be not too onerous on small projects. Perhaps, 
that is why the only project selection guidance that KSC procedures provide is a “Feasibility 
Check List” (KDP-KSC-P-2755). 

Although components of returns, costs, and risks are mentioned, the “Feasibility Check List” 
provides is no guidance on the metrics to be used. Even familiar issues of cost/ benefit 
measurement [e.g., valuation of (a) facility costs, (b) KSC FTEs, (c) multi-year cash flows] are 
not addressed. The only prescribed feasibility criterion seems to be availability of adequate 
funding. KSC procedures do not address issues of 

(a) Comparing costs and benefits that may be in different units of measurement, 

(b) Minimum expected return on investment, 

(c) Establishing the desirability of one project over another 

(d) Assessing synergies between different projects. 

Thus, even decisions about large-scale projects involving millions of dollars of investment must 
be based on the limited data and guidance the required Feasibility Check List provides. 

I recommend that KSC should develop different processes and criteria for small, medium, and 
large projects. Particularly for selection among large projects, provide better metrics of benefits 
and costs, some guidance on how to resolve classical cost benefit analysis issues, how to 
establish relative desirability among projects, and how to seek an optimal portfolio of projects. 

4. A CRITIQUE OF CMAT/ PAT 

CMAT/PAT is still evolving. This critique is based on CMAT/PAT documents here should be 
taken not as a “judgment” of CMAT but as an aid to further development of the tool. 

With the primary goal of assisting the Strategic Planning Process at KSC, CMAT promises to: 

-Use Fuzzy Logic to generate strategy ideas, to predict strategic performance, and to provide 
strategic decision support. 

- Adapt Markovitz’ Portfolio Theory for strategic decision analysis in support of a well 
rounded, balanced R&D portfolio selection. 

-Analyze the risks (i.e., the variance of the returns) of individual R&D projects as well as that 
of a portfolio of projects by using “multiple managerial inputs.” 

-Use system simulation to provide predictive analysis of STC change evolution and the “gap” 
between KSC’s technology targets and projected accomplishments. 

Roadblocks in the attainment of CMAT/PAT Objectives 

While CMAT’s goals are extremely worthy, they may not be actually attained due to: 

I. Inadequacy of available data. 
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II. Ambiguous, overlapping, internally inconsistent, and incomplete definitions of required 
data and metrics. 

III. Lack of work on the specification of a good data collection process. 

IV. Lack of work on the promised decision support models Inadequacy of Data 

Although the CMAT project is already in Phase If the documents I have examined do not 
outline precisely what data KB SI would need to build its Fuzzy Logic engine, its System 
Simulation model, or its Portfolio Analysis Tool and precisely how KB SI would go about 
collecting the necessary data. 

Instead, as requirements of the CMAT/ PAT project KBSI has accepted to 

(a) Use existing NASA data sources and systems, and 

(b) Minimize new data entry (“minimally intrusive”). 

Now, I appreciate why Spaceport Technology Development Office (STDO) may have imposed 
these requirements. Scientists and researchers are typically more interested in satisfying their 
curiosity and investigating interesting phenomena rather than thinking about potential costs, 
benefits, and risks of their research. They hate to comply with existing data collection processes 
and will surely find any additional data requirements as burdensome. Secondly, NASA’s 
engineering/scientific culture emphasizes expertise within well-defined, narrow fields. This 
culture underestimates the true value of a well-designed, broad information system that supports 
organizational learning and improved decision-making. Third, even if new data requirements are 
defined, STDO personnel may have to estimate and enter those data themselves since they have 
no authority to enforce project managers’ compliance with the new requirements. 

Unfortunately, existing NASA data sources cmd systems are woefully inadequate for the kind of 
analysis CMAT aims at. For instance, to develop its “Fuzzy Rules” CMAT would require current 
and targeted ratings (Low, Medium or High) on such factors as “Design and Engineering 
Capability,” “Science Enabling Capability,” “Safety,” or “Virtual Collaboration.” CMAT would 
not only need such rating data (albeit “fuzzy”) over several years, it would also need data on the 
correlations between the ratings of various factors. Similarly, PAT’s portfolio variance analysis 
requires either “multiple managerial input” or other means of capturing the standard deviation of 
projected benefits of individual R&D projects, as well as the co-variances of benefits from 
different projects (i.e., data on the synergies between projects). NASA’s existing data systems 
have not been collecting such data. 

Poor Data Definitions 

CMAT’s Definition of “Return,” as an example, 

o Does not value possible one-time benefits of an R&D 
o Does not value knowledge contributed by a failed project 

o Does not specify how to account for time value of money multi-year benefit flows 
o Differs from NASA’s standard list of benefits on KDP-KSC-F-2755 
o Is subject to several different interpretations and hence vulnerable to intentional or 
unintentional fudging 

o Does not allow for negative rating any of the criteria 
o Does not specify the process of arriving at the weights for different criteria 
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Similarly, CMAT’s Definition of “Risk” 

o Attempts to use an approach that is logically unacceptable, namely to take a weighted 
average of 

(a) costs to TRL9, 

(b) probability of technical success, and 

(c) the degree of collaboration. 

I do not know if “Cost to TRL 9” would necessarily increase or decrease a project’s 
probability of failure. However, it is reasonable to assume high levels of “Collaboration” 
may reduce that probability. However, to capture that relationship I would not use a 
weighted average approach. 

o Is inconsistent with CMAT’s definition of “return.” CMAT defines returns in terms of 
“benefits to NASA if the project is successfully implemented,” CMAT should not 
simultaneously say that “Risk as defined here is more from a technology standpoint 
rather than from an operational standpoint.” To be consistent, either the definition of risk 
or the definition of returns must be modified 

CMAT has not clearly defined “cost” yet But it must soon confront classical cost measurement 
issues such as: 

o Valuation of existing/ new facilities necessary for the project 
o Valuation of NASA FTEs 

o Valuation of multi-year costs (that may or may not be incurred) while approving a single 

year’s funding 
o Time value of money 

Lack of Specification of a Good Data Collection Process 

In addition to providing clear and sound data definitions, KBSI needs to outline a data collection 
process that is not too burdensome on the project managers and yet is capable of capturing 
relevant data with adequate accuracy in a timely manner. Among other things, the process should 
specify: 

o Who is to provide what data to whom at what time? 

o Who is to verify the reliability, validity, and/or accuracy of the data? 

o In the case of subjective ratings, who is to be considered an expert? Who would ensure 
that the expert is not biased; and how? 

o Who has the authority to enter or modify a project’s status data (e.g., funded/ rejected, 
successful/ unsuccessful, actual benefits generated, costs incurred, etc.)? 

o Who would have the responsibility to maintain the database over the years? 

o How will the data be statistically analyzed to obtain historic success rates, average 
benefits, costs, etc? And, how will that statistics be used to update CMAT (including its 
data collection process) and improve KSC’s portfolio selection methodology in the 
future. 

In designing the data collection process, an important principle to keep in mind is that the costs 
of data collection must be justified by the benefits resulting from the collected data. Thus, for 
projects involving relatively small investment, the data requirements must be considerably 
simpler and smaller than the data requirements for projects involving millions of dollars of 
investment. 
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At the same time, it should be realized that, in the long run, the benefits of informed decision- 
making and management practices resulting from the data collected using a properly designed 
process often outweigh the costs of that process. Particularly in the R&D context, it should be 
realized that as a project proponent prepares to provide the data required by a well-designed 
decision-making system, often, he/she is forced to improve the very design of the project so as to 
make it more acceptable in terms of the organization’s decision criteria. Often, such improved 
project designs themselves pay for the costs of the data collection system. 

Lack of Work on Decision Support Models 

At present, CMAT does not provide a clear and sound method of recommending an “optimal” 
portfolio of projects. It only attempts to show the costs, risk and returns of a given portfolio, and 
the diversity that portfolio reflects in terms of TRL levels or STDI area, etc. In other words, at 
present CMAT is simply a presentation software. 

While this presentation software is non-intrusive and supportive, it would not prove to be 
practical decision support tool. An analyst using CMAT would have to input each possible 
portfolio, print its results and then manually compare all the printed results to identify the most 
preferred portfolio. The trouble is that if one wanted to select a portfolio consisting of 10 out of 
40 projects, there are over 800 million possible alternatives! 

Personally, I would recommend the use of a generic constrained optimization model (such as the 
one described by Martino, 1995, Ch. 5) adapted separately to two sets of projects 

Set 1: Those with beginning TRL 1 to 3 and ending TRL < 4 (applied research) 

Set 2: Those with beginning TRL 4 to 6 and ending TRL< 7 (development) 

In my vision, the project selection criteria and the data requirements for Set 1 projects would be 
far simpler than those for Set 2 projects. 

In short, while CMA T’s stated goals are laudable, they are unlikely to be attained unless both 
NASA and KBSI work at removing the four major roadblocks identified above. Otherwise, with 
all the charts and diagrams it can produce, CMAT would prove to be an impressive presentation 
software. However, it would hardly be worthy of being called, “Change Management Analysis 
Tool.’’ 


5. CONCLUSION/ RECOMMENDATIONS 

KSC should recognize that the “one size fits all” nature of its current project management 
procedures is not conducive to prudent R&D portfolio selection. KSC must work on suitably 
modifying those procedures and the criteria used for project selection. 

KSC should also recognize 

(a) The inadequacy of its existing data sources and systems, and 

(b) The true value of a properly designed data collection system for R&D projects 
and remove the requirement on CMAT to use only exiting data sources and systems. 

KBSI and KSC should together come up with data definitions that are clear, theoretically sound, 
internally consistent, and complete. 

In consultation with KSC/ STDO, KBSI should outline a data collection process that is not too 
burdensome on the project managers and yet is capable of capturing relevant data with adequate 
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accuracy in a timely manner. The designed process must ensure that the costs of data collection 
are justified by the benefits resulting from the collected data. This calculus should recognize 

(a) The long term the benefits of informed decision-making and management practices, and 

(b) The benefits of improved project designs 
resulting from a properly designed data collection system. 

If KSC truly desires to bring about a strategic change in its R&D endeavors, the Center must be 
willing to change its data culture. This includes giving adequate authority to STDO to be able to 
collect the necessary data. The concern for not overwhelming researchers and developers with 
demanding data requirements must be balanced against organizational need to make informed 
decisions in support of its strategic goals. 

Finally, KBSI should develop and validate theoretically sound yet practical models that will truly 
support KSC’s decision-making, be it in the area of project portfolio selection, or predicting 
strategic performance, or identifying technology gap. KSC should work towards a full realization 
of CMAT’s stated goals by using such rigorous models. 

I am pleased to note that STDO/ KSC intend to follow up on many, if not all, of my 
recommendations . 
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